Setting Incidents: incident_key in EventsV2 API

#1

Hi PD community,

Iā€™ve just picked up a project where Iā€™d like one of my apps to trigger events and resolve alerts based on thresholds the app gets from looking at devices. However, Iā€™m struggling to find how when hitting eventsv2 endpoint (https://events.pagerduty.com/v2/enqueue) the incident_key in the incident can be populated at time of creation? No matter where I put it in the parameters or payload, it always ends up as ā€˜Noneā€™ in the subsequent incident. All the alerts subsequently created under incident contain the dedup_key, but, I want to be able to uniquely identify the incident based on my apps dedup_key I specify, without having to recurse all incidents associated alertsā€¦

Heres an example taken from the class Iā€™ve written - can you tell me where Iā€™m going wrong?

    def trigger(self, description, hostname, details, dedup_key=None):
        data = json.dumps({
            'routing_key' : SERVICE_KEY,
            'event_action': 'trigger',
            'incident_key'   : 'my-identifiable-incident-key',
            'dedup_key'   : dedup_key,
            'payload'     : { 'summary'       : description,
                              'custom_details': details,
                              'source'        : hostname,
                              'severity'      : 'error',
                            },
            'client'     : '%s running on %s' % (sys.argv[0], os.uname()[1]),
            'client_url' : 'https://my.super.awesome.app.net',
        })
        r = requests.post('https://events.pagerduty.com/v2/enqueue',
                          headers=self.headers,
                          data=data,
        )
        return r

When setting dedup_key in the above, it is the dedup_key in the associated alert. But incident_key is always None in the incident. If I chose to POST ā€˜incident_keyā€™ in the ā€˜payloadā€™, incident_keyā€™ never ends up in the actual ā€˜incident_keyā€™ field when pulling back the incidents againā€¦

Iā€™ll happily use another (appropriate) customisable field if it can make it into incident at time of creation, but Iā€™m wondering why this is the way it isā€¦?

Cheers!
Sandy

#2

Hi Sandy, to clarify with Alerts enabled on a service itā€™s expected that the incident_key will be null. This is by design, with the thinking being that (with Alerts enabled) Incidents are a ā€œcontainerā€ for Alerts, and donā€™t have an incident/dedup key of their ownā€”rather itā€™s Alerts that get deduplicated.

One approach would be to turn off Alerts (on a Serviceā€™s edit page under Incident Behavior), in which case you can expect to see an Incidents incident_key populated again.

#3

Hi Anton,
Thanks for your response, that explains the behaviour Iā€™m seeing.

In my case, I have a few integrations which together make up the service. I only want each integration to manage their own events in the service. I was hoping to identify by something in the list of incidents I return for that service, which integration was responsible for raising that incident. Does it follow then, that for each of my integrations, I must also recurse the alerts for each incident in the service, before I can find something unique to the integration that raised it (aside from putting something in the description string for example) ?

Sandy

#4

Hi Sandy, using Events API v2 with Alerts enabled on the service, an Alertā€™s ID or dedup_key will be the best way to target a specific Alert.

#5

Sandy, you can find an incident by the events dedup_key by sending a get request to this uri:
https://api.pagerduty.com/incidents/?incident_key=$dedup_key"

replace $dedup_key with the key you used or got from the events API V2 request.

#6

Hey @sanjmonkey, are you actually using alert / incident merging?

Be careful with setting the dedup_key (for alerts) arbitrarily, as alerts with a matching dedup_key will be merged into a single incident (which will hold the same description/title as the initial alert). Obviously it also holds that using a unique dedup_key, even if itā€™s identifiable - e.g. sourcea_dhejh78 where sourcea_2q1 is your identifiable / fixed part and dhejh78 was the random jibberish for that specific alert.

If you never want this to happen, flip to the old behaviour of incidents only. Same rules would then apply on the incident_key.

Alternatively, have you considered implementing a separate Service for each Integration? Thereā€™s no limit to how many Services you can have and if theyā€™re all bound to the same Team the visibility in the Incidents page will be much the same (you can even see which Integration-Service it came from in that list, which you canā€™t otherwise), and Escalation Policies are reusable.

Cheers,
@simonfiddaman